Skip to content

feat(web): keep the SDK's radio data in browser storage - #1502

Draft
danditomaso wants to merge 49 commits into
meshtastic:mainfrom
danditomaso:feat/sdk-storage-w1
Draft

danditomaso wants to merge 49 commits into
meshtastic:mainfrom
danditomaso:feat/sdk-storage-w1

Conversation

@danditomaso

Copy link
Copy Markdown
Collaborator

Description

Stores the SDK's radio data (nodes, config and channels) in a local SQLite database in the browser, and fixes issues found while testing on a real radio.

Related Issues

Depends on #1501 and meshtastic/meshtastic-sdk#216.

Changes Made

  • SDK storage kept in the browser's private file system, falling back to memory if it can't open
  • HTTP connections take about 4 s instead of 48 s
  • Sidebar no longer crashes when a connection drops
  • No toast after disconnecting

Testing Done

Unit tests pass, and connections were tested against a Heltec V4 over serial, HTTP and Bluetooth.

Screenshots (if applicable)

No visual changes apart from the removed disconnect toast.

Checklist

  • Code follows project style guidelines
  • Documentation has been updated or added
  • Tests have been added or updated
  • All i18n translation labels have been added (read
    CONTRIBUTING_I18N_DEVELOPER_GUIDE.md for more details)

https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5

Baseline typecheck: 176 errors, all pre-existing jest-dom matcher types in
test files. Unchanged by this commit.

- components.json: new-york, tsx, no rsc, css at src/index.css, lucide,
  aliases utils -> @core/utils/cn, ui -> @components/shadcn (components/ui
  would collide with components/UI on macOS).
- pnpm dlx shadcn@latest add drawer: generated components/shadcn/drawer.tsx
  and added vaul. The CLI misread the utils alias as an npm package and
  installed "cn"; removed it and pointed the import at @core/utils/cn.ts.
  Formatted with oxfmt.
- Tokens: shadcn neutral light/dark OKLCH set, mapped through @theme inline.
  Raw tokens are scoped to [data-shadcn-scope] (dark on [data-theme=dark])
  so existing screens are untouched. --accent/--accent-foreground omitted
  because the app maps --color-accent: var(--accent) for ring-accent;
  --radius-* omitted because it would override Tailwind's rounded-* scale.
- Toast.tsx: removed never-resolving token classes (border-muted/40,
  text-foreground/50, ring-ring ...) that would otherwise start resolving
  and shift the cascade. Rendering is identical.
The drawer is an overlay over whatever page is showing. Its open state
and view (list | add | edit:<id>) live in the app store as
connectionsDrawer with openConnections(view?), closeConnections() and
setConnectionsView(); nothing is encoded in the URL. connectDialogOpen,
which nothing read, is replaced by it.

- Mounted once in App.tsx next to ConnectingOverlay (drawer z-40, overlay
  z-50 stays above). With no device the drawer is forced open: no close
  button, dismissible=false, Escape prevented.
- / no longer renders Connections; the /connections page route is now a
  redirect to / that calls openConnections().
- DeviceInfoPanel, LockdownGate and the command palette's Connect New Node
  call openConnections() (the palette asks for the add view).
- pages/Connections/index.tsx moved to components/ConnectionsDrawer/
  ConnectionsList.tsx; useConnections/utils stay where they were.
- Tests: store actions, /connections redirect, open/close changes no route,
  no in-app navigation to /connections, forced vs dismissable drawer.
…queries

- Add connection is a view pushed inside the drawer (list | add in the app
  store), not a modal. AddConnectionDialog's body moved to
  ConnectionsDrawer/AddConnectionForm.tsx with its reducer, panes and
  validation intact; the Dialog wrapper and open/onOpenChange are gone and
  its casts replaced by a TabKey guard, instanceof Error and the typed
  navigator.serial API. Back button; focus goes to the view title on push
  and back to the Add connection button on pop.
- List restyled with shadcn tokens: rounded-xl tonal cards with hairline
  separators, bg-primary primary action, overflow menu via the kit
  DropdownMenu restyled with token classes (data-shadcn-scope on the
  portal). Layout is container-query driven (@sm/@md/@lg/@xl/@3xl): 1-3
  columns, avatar/padding/title step up, status moves inline with the name.
- WCAG 2.1: all text on the rem scale, nothing below text-sm except none;
  inputs text-base; no fixed heights on text containers (h-auto min-h-11);
  secondary text wraps instead of truncating; light --muted-foreground
  lowered to oklch(0.5) for 5.5:1 on --muted (stock 0.556 was 4.34:1);
  amber/green helper text uses the 700 step (4.8:1). textSizes.test.ts
  guards arbitrary px sizes. Kit Button gains a React 19 ref prop.
- SupportedBadge (white on red-500, 3.8:1) replaced by a token pill and
  removed; the empty Dialog/AddConnectionDialog folder is gone.
- Desktop (md+): a left-edge handle, role=separator with aria-valuenow/min/
  max, 12px hit area straddling the edge, grip visible on hover, focus and
  while dragging. Width clamps to 360px … min(960px, 75vw) and re-clamps
  on window resize. Arrow keys move 16px, Shift+Arrow 64px, Home/End go to
  the limits, Enter or double-click resets to 440px. Dragging previews the
  width and stores it on release; the handle stops propagation and carries
  data-vaul-no-drag so vaul's drag-to-close never starts.
- Below md the drawer is min(100vw - 16px, 440px) and not resizable.
- Persisted through the repo's useLocalStorage under
  meshtastic:web:sidebar:width; the deserializer accepts only finite
  numbers, anything else falls back to 440 and storage errors fail soft.
- Tests: restore, invalid values, clamp and re-clamp, keyboard, drag,
  limits, propagation, double-click reset.
Choosing Delete from a row's overflow menu swaps that row's actions for an
inline strip: "Delete {name}?", Cancel and a destructive Delete. Focus
lands on Cancel as the menu's focus scope unmounts (onCloseAutoFocus);
Escape cancels and returns focus to the menu trigger, and the drawer's
onEscapeKeyDown ignores Escape inside a strip so it does not close. Only
one strip is open at a time. After the delete, focus moves to the next
row's menu button, else the previous one, else Add connection.

The AlertDialog kit component and @radix-ui/react-alert-dialog had no
other users and are removed.

Tests: strip swap and focus, Escape, single strip, focus after deleting a
middle, last and only row.
packages/sdk-kmp is a Gradle Kotlin/JS build (wrapper committed) over
org.meshtastic:sdk-core-js, pinned once in gradle.properties and resolved
from mavenLocal until meshtastic-sdk#215 is released. Its jsMain holds the
@JsExport facade (createHttpClient, MeshtasticClient, ConnectionStateJs)
and an HTTP RadioTransport with Local Network Access handling. The build
emits ES modules plus sdk-kmp.d.mts into dist/, which the workspace
consumes as @meshtastic/sdk-kmp; apps/web predev/prebuild hooks build it
when missing, and CI sets up JDK 21 with a Gradle cache. Nothing in the
app uses it yet.
With localStorage meshtastic:web:kmp=1, HTTP connections are opened by
@meshtastic/sdk-kmp: the Kotlin SDK runs the handshake, owns the
connection state and reconnects (AutoReconnectConfig maxAttempts=10),
and handles Chrome Local Network Access (denied is terminal, prompt only
from a gesture). core/kmp presents the SDK client as the TypeScript SDK's
Transport so the existing MeshDevice keeps feeding the stores, maps
ConnectionState onto the saved-connection status (reconnect attempt
n/max) and classifies connect failures by the SDK's exception kinds. The
SDK bundle is a lazily imported chunk (231 KB gz).
HTTP connections always go through @meshtastic/sdk-kmp now; the TS
TransportHTTP path, the web-side handshake retry (reopen with backoff),
the pending-retry bookkeeping and the store's connectionPhase are gone.
The SDK's own heartbeat and liveness watchdog replace the TS heartbeat
for connections it owns. The connecting overlay and the drawer's status
pill show the SDK's reconnect attempt n of 10, and useDeviceConnected
reads the device status the SDK drives instead of a web-side phase.
… SDK

WebSerialTransport and WebBluetoothTransport implement RadioTransport in
jsMain. They never prompt: the web client passes the SerialPort or
BluetoothDevice it picked inside a click handler, and every automatic
connect or SDK reconnect reopens through getPorts() (matched on the saved
USB ids) or getDevices(). OS disconnect events become recoverable
transport errors so the SDK's auto-reconnect runs. Serial framing comes
from a small FrameAssembler over the SDK's WireFraming constants, since
SerialFrameAssembler and BleTransport are not in the published js klib.

Adds a Node test target with 21 Kotlin tests.
useConnections now opens all three connection types through the facade
(createHttpClient / createSerialClient / createBluetoothClient); the
Kotlin SDK owns the transport, handshake and auto-reconnect, and the TS
MeshDevice is fed through kmpTransport as HTTP already was.

Gesture rules: automatic connects pass only the saved USB ids or
Bluetooth device id and the SDK reopens via getPorts()/getDevices();
requestPort/requestDevice run only from clicks (the add-connection form
and the Connect button when no permitted device matches) and the picked
handle is handed to the facade. Picked ids are saved on the connection.

core/connections/browserDevices.ts replaces transports.ts (no casts);
AddConnectionForm passes the picked SerialPort along with the
BluetoothDevice. apps/web no longer depends on transport-web-serial or
transport-web-bluetooth.

Tests: browserDevices helpers, useConnections over fake navigator.serial
/ navigator.bluetooth (no prompt on automatic paths, picker only on
clicks, missing API, drop -> reconnecting), and the real Kotlin facade
driven end to end over fake ports/devices with a FakeRadio that answers
the handshake (connect, handed-over handle, unavailable, unsupported,
unplug -> reconnecting -> replug -> connected, GATT drop -> reconnect).
Those run under vitest's node environment: kotlinx-coroutines picks the
window.postMessage dispatcher under happy-dom, whose MessageEvent.source
never matches window, so a burst of more than 16 dispatches wedges it.
sendText, sendPacket, requestNodeInfo and requestPosition now return the
packet id straight away with the outcome as a promise, so the web client
can file an optimistic message under the id the routing ack will name.
Admin dispatch forwards lockdown_auth. The factories take an optional
StorageJs, a host-implemented StorageProvider/DeviceStorage over encoded
protobufs; ConfigBundleJs carries deviceUiConfig for the round trip.
Moves the message, draft, node and telemetry repositories, the DB opener,
migrations and the multi-tab coordinator from sdk-storage-sqlocal into
apps/web/src/core/storage with local Message, Node and TelemetryReading
types, cast-free, and drops the sdk-storage-sqlocal dependency. The
Types namespace (ChannelNumber, Destination, PacketMetadata,
DeviceStatusEnum) is now local in core/types.ts.
…untime

A MeshSession over the facade decodes the SDK's FromRadio and MeshPacket
streams into typed events and owns the web's views: device (status,
identity, metadata, lockdown, region presets), nodes (the SDK node DB
plus PKI and routing errors, favourite/ignore/remove/reset), channels
and beacon offers, chat (messages, drafts, unread, send/resend with the
SDK's packet id and delivery state) and telemetry history, all over the
web's sqlocal repositories keyed by node number. The sdk-react hooks are
replaced by same-named hooks in core/kmp/hooks over a web MeshRegistry;
subscriptions.ts feeds the device store from session events;
useConnections creates the session before the SDK connects, so there
is one handshake. kmpTransport, the TS MeshDevice/MeshClient, the TS
MeshRegistry and the sdk, sdk-react and transport-http dependencies
are gone from apps/web; Types, withLocalNetworkTarget and the Mesh
Beacon join helpers are local.
SqlocalDeviceStorage implements the facade's DeviceStorageJs (node DB,
config bundle, channels, heartbeats, session passkey) in the same
database as the web's message, draft, node and telemetry history,
keyed by node number like them; schema v5 adds the sdk_* tables and a
node_info column while the TypeScript SDK's node rows stay readable as
NodeInfo. The three facade factories receive it per connection. A
migration test proves rows written by the old storage, under the same
keys, stay readable.
Until meshtastic/meshtastic-sdk#215 publishes js artifacts the SDK
exists in no Maven repository, so CI and clean checkouts could not
build @meshtastic/sdk-kmp. vendor/meshtastic-sdk is now a git
submodule pinned to that PR's head and settings.gradle.kts includes it
as a composite build, substituting org.meshtastic:sdk-core and
sdk-testing with its :core and :testing projects; mavenLocal() is
gone. The build script initialises the submodule when it is missing,
every workflow that builds sdk-kmp checks out submodules, and the
READMEs document the one-line switch to the Maven Central release.
Connecting again on a connection whose client is still live, or on
another connection while one is live, now disconnects that client
first, so a serial port or HTTP poller is never held by a session the
app no longer reads. A reconnect cycle that the SDK reports as a lost
Local Network Access permission ends at once: denied is a terminal
error with the site-info guidance, a dismissed prompt leaves the
connection disconnected until the next click. The SDK's storage and
the web's repositories now share one open outcome, so when the sqlocal
database fails or times out both run in memory and the connect still
goes through. A lockdown handshake cycle no longer races a user
Disconnect: a closed session skips the reconnect half of the cycle and
disconnect waits for the cycle to settle.
A poll failure with the Local Network Access permission revoked is a
non-recoverable transport error, so the SDK stops instead of retrying
ten times against a permission only the user can restore. The serial
transport removes its navigator.serial listeners on disconnect and on
a failed connect; each connect used to add a pair for the page's
lifetime. Every transport discards the frames the old link queued when
it disconnects, so a reconnect's handshake starts clean. Bluetooth
reads and writes take turns behind one mutex, because Chrome rejects
overlapping GATT operations on a device.
A drop the SDK does not retry is reported by its cause rather than as
ten exhausted attempts. A click on a saved serial or Bluetooth
connection with no held handle opens the picker first, filtered to the
saved ids, so the gesture is still live; automatic paths keep
getPorts/getDevices. The node repository upserts in one multi-row
statement per chunk, and the web no longer writes node rows the SDK's
storage already writes (removals and resets remain). The connection
probe skips a local HTTP target while the Local Network Access
permission is still prompt, so opening the drawer never prompts. Dead
code is gone: MeshRegistry.has, SqlocalDeviceStorage.currentScope,
createSqlocalDb, and the facade's sendRaw, nodes, configBundle,
channels, onConfigBundle and onChannels, mirrored in FakeKmpClient.
The drawer's status probe already resolves the one permitted serial port
or the known Bluetooth device for each saved connection. Hand that handle
to the click path so Connect after a reload opens it directly instead of
showing the picker for a device the browser already allows.
A held serial port or Bluetooth device is forgotten when the drawer probe
no longer finds it or the SDK reports it unavailable, so the next Connect
click reaches the picker again. Connecting resolves the picker before
tearing down the live radio, keeping the click's activation for the prompt.
… logs

A failed write on a lost USB link made the Web Serial and Web Bluetooth
transports report a recoverable Error and then throw. The engine treats a
throwing send as fatal: it queues Disconnect ahead of the state change,
cancels the session without retrying and ends in `disconnected`. The web
subscriber ignored a `disconnected` that no `reconnecting` preceded, so the
UI stayed on "Connected" with nothing logged. The same silent path serves
the engine's 60 s liveness timeout.

- Transports report link loss only through state; send() no longer throws.
- The OS disconnect handler releases the lost port before forgetting it.
- A `disconnected` the SDK initiated marks the connection as an error with
  a message and an error log line.
- `setSdkLogger` routes the SDK's LogSink and the facade's connection-state
  transitions into the web's tslog "sdk" logger.
- The serial fakes follow Chromium: the same SerialPort object survives an
  unplug, both streams fail with a NetworkError, open() fails until the
  replug, close() on a closed port throws InvalidStateError.
While the SDK reconnected, the drawer's refresh probed the connection and
overwrote its "connecting" status with the probe's "online", so the row
offered Connect instead of "Reconnecting… n of 10". A click then started a
second client against the same port: the live session was torn down and
the new one, whose connect fails on the busy port, stayed registered as
the active session. The app then read a session that never saw a
handshake: no own node, no channels, only the persisted node list.

- refreshStatuses leaves connections with a live session alone.
- connect() is a no-op while that connection's session is reconnecting.
- A session whose connect failed is unregistered, not left as the active one.
- The session's own handshake cycle (after an unlock) no longer reports
  its planned disconnect as a drop.
- The facade e2e asserts the session's own node and channels survive an
  auto-reconnect; the fake radio can report extra nodes and channels.
The second tab shows an overlay instead of mounting; Use here asks the
owner to disconnect and release the app lock, stealing it after 3s.
… lost

Fresh serial connects log "undecodable FromRadio frame" during Stage 1 and
sometimes leave the sidebar on "Loading…": the own node never resolves. The
facade's frame hand-off is a copy at offset 0 (copyOfRange slices, the
Uint8Array view is over that copy), so the web and the engine decode the
same bytes; a frame that fails in the web fails in the engine too and the
engine recovers through its retries while the web kept a single-frame
dependency for its own node.

- The facade exposes `myNodeNum` from the handshake's my_info (unsigned)
  and `ownNode` as the SDK holds it.
- On `connected` the session adopts both when the frame stream left them
  unset, and fills the own node's user from the get_owner response the SDK
  fetches at Ready.
- Tests feed frames as views into a larger buffer, several per chunk and
  split across chunks, and assert the bytes reaching JS decode.
Gating the app behind the tab lock moved enableMapSet() from the first
render to whenever the lock is granted. The device store's IndexedDB
rehydration ran before that in a second tab (and in any tab that gets the
lock late), produce() threw on the devices Map, the persisted devices
were never rebuilt through deviceFactory, and the first Connect hit a
method-less device: "device.setConnectionId is not a function", then
DialogManager reading dialog.QR of undefined.

The store enables Map support itself when it loads. The tab-lock gate
also applies the stored theme before its first paint, so the overlay in a
fresh second tab follows the user's theme.
…calls

After "Use here" the old owner disconnected its sessions and released the
app lock but kept the sqlocal worker, so its OPFS handle stayed with that
tab. The new owner's Connect then stalled on storage the SDK touches
during connect, and nothing bounded those calls.

- disconnectAllConnections closes the sqlocal database after the sessions
  end and forgets the singleton, so the handle moves with the lock and
  this tab reopens cleanly if it takes the app back.
- The SDK's storage adapter answers every call within 5 s: a read falls
  back to its empty value, a write counts as done, each with a warning.
- Tests: takeover closes the storage; a database that never answers still
  lets connect proceed.
Review findings I1, M1 and M2 on the single-tab lock.

- I1: a connect still choosing a port or opening storage when the app
  was handed over went on to open the serial port and the database in
  the old tab. disconnectAllConnections bumps a handover counter; connect
  checks it after every await and abandons, and getStorageDb refuses to
  reopen until this tab owns the app again (resumeStorageDb on grant).
- M1: a late grant of the normal request arriving after the steal timer
  made the claimant steal from itself and then tear itself down. Each
  request carries a grant token; only the current grant's loss counts,
  and a grant that lands once stealing began is let go at once.
- M2: the gate rendered nothing while the lock was still being asked
  for, so an early click hit a blank page. The overlay now renders with
  Use here disabled during that time and takes focus once it is enabled.
- M3 (retry a busy serial open after handover) is already covered: the
  transport retries a busy open three times over 1.5 s.
Every fresh serial connect decoded a few garbage frames during Stage 1:
the device's queued output (log text, the tail of an earlier frame)
reaches the framer first and a chance 0x94 0xC3 inside it is taken as a
header. The SDK's jSerialComm transport purges its input buffers before
sending the wake bytes; Web Serial has no purge, so the transport now
reads and discards until the line has been quiet for 50 ms (300 ms at
most) before reporting Connected and starting the framer. The web logs a
frame it cannot decode at debug, as the old transport did; the engine
still reports its own decode failures.
Delete packages/sdk, sdk-react, sdk-storage-sqlocal, transport-* and
tests/utils. The web app runs on the Kotlin SDK facade (packages/sdk-kmp).
Update root package.json, pnpm-workspace.yaml, lockfile, CI and release
workflows (release loop skips private packages; ui marked private),
README and TESTING.md.
Every MeshPacket reached the web twice: as a raw FromRadio frame the
session decoded and ignored, and re-encoded through onPacket. The tap
now reads the frame's field tags and skips frames carrying a packet,
so each packet is decoded once on the web side. Frames are no longer
copied when nothing listens.

Claude-Session: https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5
Each node change copied the whole node map and notified every reader,
so a 500-node handshake made 500 copies and renders and every heard
packet copied the map again. Changes now land in one draft map that
is published in a microtask; reads see pending changes immediately.

Claude-Session: https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5
Points the submodule at the storage-sqldelight js target (cb9d17b). The
commit exists only locally until PR1 is pushed; move the pin to the
pushed sha before this branch leaves the machine.

Claude-Session: https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5
NodesClient writes each published batch of changed nodes, and every node
of a snapshot, to its repository with one upsertBatch so a later session
hydrates the list it last saw.

Claude-Session: https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5
Once any bounded store degrades or activation falls back, later
activations go straight to memory without touching the inner provider.
The warning is logged only when the degraded flag flips, an activation
failure names its cause, and build --if-missing also rebuilds when the
SQLite worker core is missing from dist.

Claude-Session: https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5
oxlint cannot resolve the default export of a ?worker&url import, so
import/default is off for browserStorage.ts only. A new test proves the
SDK storage openRepositories resolves reaches the client factory, and
the storage sentinels are plain frozen objects.

Claude-Session: https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5
The drawer already shows the connection as disconnected; the toast only
repeated it.

Claude-Session: https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5
The HTTP handshake fetched one FromRadio per round trip (~180 ms each,
~250 frames), and after a write it slept until the next 3 s poll, so a
connect took ~48 s. Poll with all=true, split the firmware's
back-to-back FromRadio messages at each payload field, and wake the
poll loop after every send. Connect now takes ~4 s on a Heltec V4.

Claude-Session: https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5
Sidebar picked its connection with useActiveConnection() ||
useDefaultConnection() || useFirstSavedConnection(), so it called fewer
hooks while a connection was active. When the connection dropped, React
saw the hook order change and the store selector read the wrong slot,
throwing "Cannot read properties of undefined (reading 'length')" into
the error boundary. useDisplayedConnection calls all three every render.

Claude-Session: https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5
Point the submodule at danditomaso/meshtastic-sdk feat/storage-sqldelight-js
(f16c8db, meshtastic/meshtastic-sdk#216, stacked on meshtastic#215) instead of a
local-only commit, and map the SDK's new SendFailure.RateLimited to
RATE_LIMIT_EXCEEDED so chat keeps showing "rate limit exceeded".

Claude-Session: https://claude.ai/code/session_01K9vyKS6Q1tGo9xUfNGCUW5
@vercel

vercel Bot commented Nov 16, 2026

Copy link
Copy Markdown

@danditomaso is attempting to deploy a commit to the Meshtastic Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant